Learning through deterministic assignment of hidden parameters

Supervised learning frequently boils down to determining hidden and bright parameters in a parameterized hypothesis space based on finite input-output samples. The hidden parameters determine the attributions of hidden predictors or the nonlinear mechanism of an estimator, while the bright parameters characterize how hidden predictors are linearly combined or the linear mechanism. In traditional learning paradigm, hidden and bright parameters are not distinguished and trained simultaneously in one learning process. Such an one-stage learning (OSL) brings a benefit of theoretical analysis but suffers from the high computational burden. To overcome this difficulty, a two-stage learning (TSL) scheme, featured by learning through random assignment for hidden parameters (LtRaHP) is developed. LtRaHP assigns randomly the hidden parameters in the first stage and determines the bright parameters by solving a linear least square problem in the second stage. Although LtRaHP works well in many applications, it suffers from an uncertainty problem: its performance can only be guaranteed in a certain statistical expectation sense. In this paper we propose a new TSL scheme, learning through deterministic assignment of hidden parameters (LtDaHP), where we suggest to deterministically generate the hidden parameters by using minimal Riesz energy points on a sphere and equally spaced points in an interval. We theoretically show that with such deterministic assignment of hidden parameters, LtDaHP with a neural network realization almost shares the same generalization performance with that of OSL, i.e., it does not degrade the generalization capability of OSL. Thus, LtDaHP provides an effective way to overcome both the high computational burden of OSL and the uncertainty problem of LtRaHP. We present a series of simulations and application examples to support the outperformance of LtDaHP, as compared with the typical OSL algorithm: Support Vector Regression (SVR) and an typical LtRaHP algorithm. The study conducted in this paper is a novel trial to tackle supervised learning problems simply and efficiently.